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A common approach to needs analysis involves the 
asvelopment cf a self^report questionnaire administered to members of 
the Intended audience* Such an approach poses a problem however^ 
eoncarn for data reliability demands polling as many respondents as 
possible, vhile concern for cost-effectiveness dictates minimizing 
the size of the sampie* The solution involves a sub-group of the 
population responding to all items in the guestionnairer and 
parameters estimated from the sample data. Set this approach may not 
hm the most effective and efficient one. Multiple matrix sampling 
(HHS) ^ a technique whereby both the population of respondents and the 
universe of items are jointly sampled^ may be the answer. Basically^ 
this technique Involves the random assignment of some items to a 
random sasiple of respondents, with other random samples of 
respdndents receiving different random items. A computer estimates 
the parameters from the sample data gathered. The study reported 
herein begins with the pool of data generated by a self*report 
queatlonnaire needs analysis and proceeds to estimate those same 
parameters through the use of MMS. These estimates are then compared 
to the known values and to corresponding estimates from traditional 
respondent* sampling techniques. The paper concludes with a summary of 
the advantages and disadvantages of MMS. (Author/BL) 
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A typical approach to identifying the training needs of a specified 
groups especially in cases where a particular set of skills or certain knowledge 
is required {as for job-related training), is to perform an analysis of the 
tasks that comprise the Job roles then to somehow measure whether or not the 
individual is capable of performing each task. Any discrepancy between the 
job requirements and the Individual's performance level 1s, ipso facto ^ the 
need for training. 

This paper provides a description of how such a task analysis was used 
recently as a base for a needs analysis. The needs analysis is then used to 
demonstrate a potential application of multiple matrix sampling, and to 
discuss the efficacy of the procedure. 

Multiple matrix sampling is* as the name implies, a sampling strategy 
with related statistical procedures which is particularly useful to needs 
analysts who employ a metric approach. Its worth as a tool for the measure- 
ment of needs lies in its capacity for Increased precision with decreased 
expenditure of time and energy (and, therefore, dollars) in comparison to 
other commonly-used sampling strategies. 

When an attempt is made to measure a variable in a number of people, 
typically a number of parallel, or alternative, measures must be used, 
Psychometry simply has not progressed far enough to allow very precise 
measurements of psychological variables using only one measurement. Usually, 
this limitation means that in order to get one answer, several questions must 
be asked. For example, to answer the question ''How competent 1s a given 
individual as a manager/adm1nistrator?''--a question, incidentally, which must 



be answered under the needs Identification model described earlier in order to 
determine whether the individual has any need for trainings-it may be necessary 
to ask a number of questions relating to ability to define staff roles 5 develop 
policy^ Interpret and implement procedures appropriate to pertinent policy or 
legislations regulate admissionSi develop and foster communication patternSi 
supervise employees, delegate responsibili ty^ and so on. In fact, to be 
technically correct, it would probably be necessary to ask several sub-questions 
to ansv/er each of the questions just identified, 

A typical means to answer a question like the example above is to 
administer a questionnaire or test of some kind to the individual whose needs 
are to be assessed, (The exact form of the data collection technique Is less 
Important than the fact that the data so collected can be adequately repre- 
sented by a score which is the sum of the Individual's responses to a set of 
questions. ) 

Unfortunately, within the real-world constraints of time and money, 
while it Is often possible to ask each person about his or her needs as an 
individual, it is seldom possible to act upon those needs for each individual, 
as an individual. In other words s when the needs of SO or 100 manager/ 
administrators are to be assessed, they are typically all asked the same 
questions, then their answers pooled to obtain a profile of the so-called 
"average" manager/administrator. The learning experiences are then designed 
on the basis of the ^'average'' needs thus derived. 

It Is in just these situations, where the focus is on a group of 
people rather than on an individual, that multiple matrix sampling displays ' 
its power. ||n 
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Sampling Strate gies 

Usually the way in which sampling is done to gather information depends 
on how many potential respondents there are. If there were only 30 people 
1n the population of concern ^ it is probable that an attempt would be made to 
gather Information from each and every one of them. This strategy is called 
census data-gathering. On the other hand, if there were 3,000 people in the 
population of concerni randomly sampling say^ 100 or 300 or 1,000 people 
(depending on the return rate we expected) might be done in order to generalize 
their responses to the population. This strategy^ is called respondent 
sampl ing . Somewhat less comprehensible in the context of needs identification 
(albeit quite comprehensible in the context of developing standardized test 
items) Is the notion of item sampling , which involves randomly sampling items 
from the total Item pool (or universe) and administering them to all respondents 
in the population* 

Multiple matrix sampling can be conceptualized as a combination of 
respondent sampling and item sampling. The application of multiple matrix 
sampling simply involves administering random samples of Items (which constitute 
sub'tests) to random samples of respondents , usually in such a way that every 
item in the universe is responded to by some respondents, and every respondent 
responds to some items. Thus (usually) all sub-tests are responded to, and 
^11 respondents respond, but no respondent responds to all sub-tests, nor do 
any sub-tests get responded to by all respondents. 

Incidentally, 1t is worth noting that neither the sub-tests comprising 
the "total" test (or item universe) nor the respondent samples comprising 
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the population need be equal in size to one another, Although 1t is frequently 
convenient to have N ? t respondents (where N = the population and t^ = the 
number of sub-tests) per sample^ and to administer K r t^ Items (where K = 
the number of items in the universe) to each sample, there 1s no requirement 
by the mathematics of multiple matrix sampling that either the item sample 
size or the respondent sample size be constant, 

Advantjges a nd Limitations of Multiple Matrix Sampl ing 

Sufficient work has now been done with multiple matrix sampling in 
both theoretical and practical applications that certain generalizations can 
be derived (Shoemaker^ 1973), Although by far the largest part of the research 
to date has been done with data from ability or achievement testing situations, 
where responses can be coded as correct or incorrect (1 or 0), at least a 
couple of studies (Loadman, 1972; Pugh, 1971) have used attitudinal or opinion 
scales as well. When considering the advantages and limitations of muHtple 
matrix sampling outlined below^ it should be kept In mind that It would be 
wise to replicate a nhmber of the achievement/ability testing studies with 
needs analysis data before generalizing into that domain. 

Nevertheless s there are indications that multiple matrix sampling in 
needs identification would have a number of advantages over other sampling 
strategies. Among these are: 

{ 1 ) the increased fe asibility of using a large questio n pool . 
Designers of survey questionnai res ^ either for needs identification or for 
other appl Icatlons t are constantly and acutely aware of the length of the 
; questionnaire, How often are Important questions unasked for fear of creating 
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an inordinately long questionnaire? Multiple iiiatriK sampling \my help i'Ucli 
problems, 

(2) reduced standard j_rror of estiniatg. In CQmparison to the 
respondent sampling schemes that are typically applied in needs identification 
surveys, multiple matrix sampling provides a greater degree of precision, as 
reflected by a reduced standard error of estimate (when the number of observations 
is kept constant). This outcome has been demonstrated both theoretically 

(Lord & Novick, 1968) and practlcany (_e,g,, six studies cited by Shoemaker, 1973). 

(3) shorter questionnaire completion time. There is a social cost 
associated with the completion of a questionnaire. Each of a certain number of 
people must expend a certain amount of time and energy in order to generate 

the information sought by the resea: cher. A researcher might tend to think of 
the length of a questionnaire only in terms of how it influences the rate of 
return of the questionnaire^ but taking a wholistic point of v1ew^ it seems 
almost unethical to waste respondents' time by asking for information that 
might be obtained more efficiently and more precisely through a modification in 
methodology. 

(4) likely more palatable t^o respondents. Return rate and ethical 
considerations aside i it could be argued that researchers owe it to themselves 
and to future researchers to make the respondents' jobs as palatable as possible. 
If respondents were to stop providing researchers with data, needs ,dentificat1on 
would become an even more difficult task than it is now* 

Multiple matrix sampling also has some limitations which need to be 
recognized and taken into account. Among these are: 
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(1) the procedure applies to groups, not to^J_nd^iv^id^ This p^per 
began by pointing out that sometimes an individual-oriented sampling scheme Is 
used when all that are wanted are group data. However^ there are times when 
there is a legitimate need for data on individuals, and in those cases, of 
course, multiple matrix sampling could not be applied, 

(2) logistics and computations . The application of a multiple matrix 
sampling scheme requires little more planning than do more conventional schemes* 
To be more specific, the Item sampling must be rationalized and performed, and 
alternate forms of the questionnaire must be printed and differentially handled 
for mailing. 

The computations involved in correcting for the fact that some of the 
respondents answer some of the sub-test are quite lengthy and complex. For- 
tunately, a computer program is readily available (Shoemaker, 1973) to perform 
the computations. 

(3) the conte xt effect may be overlooked. Multiple matrix sampling 
assumes that the respondent will answer a question in the sam?: way on a short 
questionnaire as he or she would answer the same question within the context of 
a longer questionnaire. Studies by Burton and Remer (1972), Hill (1975), Huck 
and Bowers (1972), Shoemaker (1970b), Sirotnlk (1970), and Sirotnik and Welling- 
ton (1974) have indicated that context effects are minimal. However, Novak 
(1974) and Feldt and Forsyth (1974> both found some context effects. The latter 

.study is especially interesting since it did not find a context effect for an 
English examination, but did find one for a mathematics examination. Indicating 
a possible context effect by subject-matter interaction. This finding leads to 
the speculation that differnt kinds of needs analysis might differentially show 
evidence of context effects. Further study In this 
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area of context effects * especially in needs analyses s seems warranted. 

(4) assumpti on o f random sampling. Multiple matrix sampling requires 
that both the items and the respondents be sampled randomly, without replace- 
ment. In the application of multiple matrix sampling to need identification^ 
there will be times when this assumption can be met to a greater or lesser 
degree. Apparently, the validity of application of the procedure is proportional 
to the degree to which the assumption can be met. Here again ^ empirical 
evidence on the robustness of the procedure would be welcome. 

The first and last limitation, you will note, refer not so much to 
the procedure of multiple matrix sampling itself as to the potential for its 
application in specific situations. 

Procedures Involved in M ultiple Matrix Sampling 

The procedures involved In multiple matrix sampling are quite 
straight- forward. In practice , sub-tests are created from the item pool by 
randomly sampling without replacement to generate the number of sub-tests 
desired, making certain that all items are represented in at least one sub- 
test, (Technically, there is no requirement that all items be represented, 
but it makes little sense In the needs identification context to Identify an 
item as worthy of an answer, then not attempt to get an answer for it*) The 
items may be selected under a stratified random sampling scheme, if such a 
scheme is appropriate to the situation. Each sub-test is then administered 
to a randomly-selected (without replacement) subset of the respondent 
population. The respondent sample, too, may be generated using a stratified 
approach, A number of rules of thumb.vjiae^>,Avj3a.bls CShoemaker, 1973)to guide 
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the researcher in choosing the most appropriate sampling plan to minimize the 
standard error of estimate under both normal distribution and skewed 
distribution conditions. 

The application of the multiple matrix sampling procedure will be 
illustrated through reference to the needs analysis mentioned earlier. Under- 
standing why multiple matrix sampling would be advantageous 1n this context 
requires a brief description of the circumstances surrounding the needs analys 

An Example of a Needs Ide ntification 

The population of concern was managers/administrators of social 
housing in the Province of Saskatchewan--a group of some 260 people. A task 
analysis of the job role of the social housing manager was done by consulting 
with a number of managers and their superiors to identify the various tasks 
and sub-tasks whose performance was required for the satisfactory execution 
of the manager's job. As welU var.ious job descriptions and training program 
syllabi from elsewhere were studied. The task analysis broke the managers' 
functions into seven broad categories, namely (1) board/authority relations ^ 
(2) family/relative relations^ (3) tenant/resident relationst (4) staff 
relations, (5) maintenance and security, (6) managerial/administrative 
knowledge and skills, and (7) finance/office routine, Within each area, a 
number of task and sub-task descriptions , developed by the project coordinator 
In conjunction with the manager-consultants, sought to circumscribe the 
knowledge and skills required by the manager's role. In other wordSs the 
resulting task descriptions were such that if, when taken together, they 



accurately described what a particular Individual were capable of doing, then 
it was highly likely that that individual would be an effective and efficient 
manager. If, on the other hand, the individual under consideration could not 
do most or all of the tasks described, the individual would likely perform 
poorly as a social housing manager. 

Conversion of the task analysis into a diagnostic questionnaire was 
accomplished by presenting each task and yub-task description individually and 
asking each respondent to Identify his or her competence at the task. In other 
words, the respondent was asked to react to each description by checking a 
box to indicate whether he or she felt (a) sufficiently competent 1n the 
task to not need training, (b) sufficiently lacking competence to desire some 
training* or (c) sufficiently lacking competence to definitely require training^ 
with respect to that task. 

In addition to finding out how competent each individual perceived 
himself or herself to be with respect .0 each task, the coordinator of the 
project felt it necessary to also determine how important each task was per- 
ceived to be with respect to the manager/administrator's role. Therefore, 
the questionnaire also asked each respondent to indicate to wha ^xtent each 
of the tasks and sub-tasks described was Important in terms of his or her job 
role. Specifically, the respondents were asked to Indicate whether each task 
description was (a) very important, (b) important, (c) not important, or (d) 
not applicable to his or her job. 

The task analysis Identified 204 tasks and sub^tasks associated 
with the role of the sociaT housing manager- Each task and sub-task description 
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raquired two responses, one 1n terrns of the respondent's self'-perceived 
competence vis-a-vis the task, and the other in terms of the importance of the 
task vis-a-vis the respondent's job. When these items were added to the 
demographic data and data regarding the preferred format of learning experiences 
that the project coordinator wanted to collect, the result was a very lengthy 
questionnaire. 

This researcher's lack of sufficient familiarity with multiple matrix 
sampling at the time of implementation of the needs analysis prohibited the 
application of the technique. Had greater familiarity obtained^ it might 
have been possible to overcome some of the political and temporal exigencies 
that prevented the application of multiple matrix sampling. As it waSi the 
census approach was selected, and the lengthy questionnaires were sent to all 
members of the client population. 

Despite some evidence by Burton and Remer (1972) ^ and Champion and 
Sear (1969), that the return rate of questionnaires is not adversely affected 
by questionnaire length, it is difficult to believe that some differential 
self-selection on the part of respondents does not occur in response to such 
extreme questionnaire length, or, at leasts that within a systems view, a 
social cost of some kind is not exacted frOT the respondent population. It 
cannot be known whether or not a shorter questionnaire would have yielded a 
higher return rate, but it is fairly certain that with the application of a 
shorter questionnaire, a certain number of person-hours' worth of time could 
have been made available for other activities. 

Nonetheless, the realities were such that the lengthy questionnaire 



P=^ - Wats used. It yielded a 54% return rate, and the data were used to determine 
training needs of social housing managers. Responses to the competence 
;£= rating were converted to numerical terms (1 - definitely need trainings 2 = 
r could be Improved, 3 ^ do not need training) ^ as were the responses Indicating 
V the Importance of the task to the respondent's Job (1 ^ very importfnt^ 2 = 
important^ 3 - not important^ 4 - not applicable). Using these item scores* 
subscale scores were computed for each of tlie seven major areas Identified by 
the task analysis (i*e*, board/authority relations ^ family/relative relations , 
tenant/resident relations s staff relations, maintenance and security, managerial/ 
administrative knowledge and sklllSs and finance/office routine) by summing 
the responses to the items comprising the subscale* 

The Efficacy of Multiple Matrix Sampling--A Preliminary Investigation 

The abundance of available research describing the efficacy of 
^multiple matrix sampling in applications to testing situations (e.g.s Cook & 
^ ; 19671 Hill , 1975; Novak, 1974; Plumlee, 1964; Pugh, 1971; 

Shoemaker^ 1970a; Sirotnik, 1970; Sirotnik & Wellington, 1974), together with 
the peculiar problems associated with the lengthy questionnaire in the current 
needs idantif ication study prompted some preliminary investigation into the 
technique. Specifically, an attempt was made to use the data generated by 
the census -samp led needs Identification in a post-mortem sense to provide an 
:a of and to help gauge the efficacy of the multiple matrix sampling 

technique. The results were not especially encouraging* However^ they do 
point to several aspects of research on the method that are needed before 
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Widespread application of the niultiple matrix sampling technique to needs 
identification can be recommended. 

In post-mortem sampling^ one begins with an extant data base* for 
which parameters are known. One then samples from that data base, pretending 
that the parameters are unknown, and estimates the parameters from the sample, 
be It respondent sample or matrix sample. Finally, one compares the estimates 
of the parameters to their known values. 

^^■^ Using the returned questionnaires as a data base, the post-mortem 

analysis compared the known parameters for two of the seven subscales with the 
estimates for those parameters using (a) a 10^ respondent sample, (b) a 20% 
respondent sample, and (c) a multiple matrix sample set. 

For each of the 10% and 20% respondent-sampling schemes, four random 
samples were obtained from the data base, and jubscale scores were computed. 
For the multiple matrix sampling scheme, each of four randomly sampled (with- 
out replacement) respondent sets were "administered" a sub-test of randomly 
selected (without replacement) items (i.e., the responses to the appropriate 
items were culled from the data base for each sub-set of respondents), and 
processed with the multiple matrix sampling computer program provided by 
Shoemaker (1973), 

Contrary to expectations, and to previous findings of a number of 
studies (e,g,. Cook & Stufflebeam, 1967; Hill, 1975| Novak, 1974^ Plumlee, 
1964; Pugh, 1971; Shoemaker, 1970a; Sirotnik, 1970; Sirotnik & Wellington, 
1974), in every case the estimate determined by the multiple matrix sampling 
scheme was more deviant from the known parameter than the corresponding 



estimate determined by the respondent sampling scheme (see Table 1), 



Insert Table 1 here, 

A couple of possible reasons for the relatively low concordance of 
the sampling schemes suggest themselves. First, the data base which was used 
for the post-mortem^ sampling was relatively small— 139 respondents. Although 
examples of applications of multiple matrix sampling are found In the literature 
for relatively small respondent groups (e,g., three kindergarten classes used 
by Shoemaker and Okada, 1970} the studies In which statistics derived through 
multiple matrix sampling are compared to those derived via other sampling 
schemes typically range upward from several hundred (e,g,s Novak, 1974; 
Pugh, 1971; Shoemaker, 1972; Sirotnik & Wellington, 1974), Shoemaker (1973) 
[lotes that although some of the original equations underpinning the multiple 
matrix sampling technique were limited to dichotomously-scored Items 
administered to reasonably large (i.e., 500) numbers of respondents i the 
equations used 1n his program are . .more generally applicable and permit 
multiple matrix sampling being used with a wider variety of sampling plans 
[p, 34]." On the other hand, Jaeger (1974) seems to imply that the technique 
ought only to be used in situations involving more than 300 units of analysis. 
Perhaps there 1s a lower practical limit with respect to the number of 
respondents for the valid application of multiple matrix sampling. Some 
empirical clarification on this point would be most welcome. 

Second, the data base used may have been idiosyncratic. Shoemaker 
(1973) notes that as the reliability of the distribution of test scores 



increases, it becomes increasingly difficult to estimate parameters* The 
. idlutlon he recommends for this problem Is obtaining a relatively large number 
.of observ Thus, this second possible explanation for the results 

observed leads in the same direction as the first. Since it would usually be 
> :txpectad that needs identification data would be highly reliable (if the task 
analyses underlying them were well done), this limitation of the multiple 
matrix sampling procedure might well interfere with Its application to all but 
large-scale needs identification studies. 

Implications 

The single example described in this paper should not be confused 
with a rigorous study comparing multiple matrix sampling to other sampling 
schemes* This study does not undo what has bean done before ; It does not 
even cast doubt on earlier studies. Rather, it points out an area of concern 
for the legitimate application of multiple matrix sampling to needs 
1dent1f1cat1on--that of the lower limit for respondent sample size in the 
context of typically high reliability data. 

The technique of multiple matrix sampling appears to hold much 
promise for needs Identification, but prior to its wide-spread application, 
some questions must be pursued. This paper ends* as do so many others* with 
a plea for further study--this time on the application of multiple matrix 
sampling to needs identification. 
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Table 1 



Efficiency of Estimating Two Subscale Means Using Various 

Sampling Strategies 



Samp 1 inS' Scheme 

■ -- 


Subscale A 


Subscale B 


Census 


37.295 


82.561 


20% Respondent Sample 

#1 

#2 
#3 
#4 

Average 


32,655 
36.821 
38.821 
41.536 


85.862 
62.571 
82.821 
77.643 


37.458 


77.532 


101 Respondent Sample 
#1 

n 

#3 
#4 

Average 


38.429 
35.929 
38.000 
40.714 


79.643 
79.286 
83.286 
70.786 


38.268 


78.250 


Multiple Matrix Sampling" 

#1 
#2 
#3 
#4 

Average 


29.405 

28.238 
28.500 
28.973 


105.360 
105.671 
107.954 
107.826 


28.779 


106.703 



For subscale five subtests of 10 items and one of 
9 items were "administered" to five groups of 23 
respondents and 1 group of 24 respondents, respectively; 
for Subscale four subtests of 4 items were "administered 
to three groups of 35 and one group of 34 respondentSi 
respectively. 
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